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ABSTRACT ARTICLE HISTORY 
Teachers can implement a high quality Daily Report Card (DRC) Received 25 October 2019 
intervention when they receive face-to-face consultation or Revised 19 April 2020 
interactive online supports. Yet, it is unclear which method is Accepted 20 April 2020 
most cost-effective. Using an ingredients-based approach and 

societal perspective, we examined costs and cost-effectiveness 

(compared to typical practice) of three implementation strate- 

gies (face-to-face standard consultation, face-to-face enhanced 

consultation, interactive online supports) with 112 elementary 

school teachers. Teachers received consultation for DRC imple- 

mentation with one student with or at risk for ADHD. Over 2 

months, we collected data on teachers’ implementation and 

changes in student behaviors. Regarding cost per student, 

enhanced consultation was the most costly ($864), followed 

by standard consultation ($634) and interactive online sup- 

ports ($307). Regarding cost-effectiveness (costs required to 

achieve the desired effect beyond typical practice), interactive 

online supports were the most cost-effective followed by 

enhanced consultation and standard consultation. We discuss 

implications for research and maximizing outcomes given dol- 

lars spent. 


In the context of multi-tiered systems of support, elementary school teachers 
are increasingly being expected to implement both universal classroom 
management strategies to prevent disruptive behavior and targeted interven- 
tions with students who need additional support. However, there is wide 
variability in teachers’ implementation of these interventions (e.g., Fabiano 
et al., 2010; Owens et al., 2017). Multiple implementation strategies have 
been used to facilitate teachers’ use of effective intervention strategies, 
including one-time workshops (Jurbergs, Palcic, & Kelley, 2010), behavioral 
problem-solving consultation which represents a best practice in school 
psychology (Kratochwill, Altschaefl, & Bice-Urbach, 2014), behavioral pro- 
blem-solving consultation with enhancements (e.g., with motivational inter- 
viewing; Owens et al., 2017; Reinke, Lewis-Palmer, & Merrell, 2008), and 
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interactive on-line technologies (Mixon, Owens, Hustus, Serrano, & 
Holdaway, 2019; Owens et al., 2019). Although studies that compare the 
efficacy of multiple implementation strategies are emerging (e.g., Bradshaw, 
Pas, Goldweber, Rosenberg, & Leaf, 2012; Owens et al., 2017), we are una- 
ware of any studies that have estimated the cost-effectiveness of multiple 
approaches. 

Cost-effectiveness analyses represent a critical factor that could inform 
resource allocation in schools. For example, school administrators often have 
to choose implementation support activities (e.g., one-time workshops; on- 
going consultation) from among options that likely vary in cost, intensity, 
and effectiveness. To be able to reduce unnecessary costs (e.g., associated 
with applying a more intensive approach than needed) and maximize the 
effects of costly resources by only applying intensive supports to teachers 
who need it, we need to understand the costs and cost-effectiveness of 
multiple strategies. This study advances the literature by comparing the 
costs of three implementation strategies among all consented teachers 
(intent-to-treat analyses) and among the subset of teachers who were actively 
engaged (completers analyses), and by examining the cost-effectiveness of 
multiple implementation strategies relative to the typical practice. 


Implementation strategies for classroom interventions 


Implementation strategies are defined as strategies, methods, or techniques 
used to facilitate the adoption, use, and sustainment of an evidence-based 
practice (Cook, Lyon, Locke, Waltz, & Powell, 2019; Proctor, Powell, & 
McMillen, 2013). In the field of school psychology, these are also referred 
to as implementation supports. There is a large body of literature that shows 
that behavioral problem-solving consultation is an effective implementation 
strategy. Some studies have examined the impact of such face-to-face con- 
sultation (relative to typical practice) when offered to teachers on a biweekly 
or monthly basis over 1 year, with (Murray, Rabiner, Schulte, & Newitt, 
2008) or without parents present (Fabiano et al., 2010; Owens, Murphy, 
Richerson, Girio, & Himawan, 2008) and with (Owens et al., 2017) or with- 
out observation and performance feedback (Fabiano et al., 2010). 

These studies document that these applications of problem-solving con- 
sultation as an implementation strategy produce adequate implementation 
outcomes and subsequent positive student outcomes (Frank & Kratochwill, 
2014; Noell & Gansle, 2014; Solomon, Klein, & Politylo, 2012). Namely, they 
are effective in producing significant improvement in teachers’ use of class- 
room management strategies (e.g., Conroy et al. 2015) and effective in 
helping teachers achieve high implementation integrity with a targeted inter- 
vention (e.g., across studies, on average teachers implemented a daily report 
card intervention on 70% to 80% of school days; Fabiano et al., 2010; Owens 
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et al., 2008). Further, problem-solving consultation is acceptable to teachers 
(Conroy et al., 2015). Notably, these applications also subsequently improve 
student outcomes; producing moderate to large changes in student behavior 
relative to baseline among students receiving the intervention (Conroy et al., 
2015; Owens et al., 2012). However, there is significant variability in teacher 
implementation integrity within each approach. For example, in the context 
of ongoing face-to-face problem-solving consultation, some teachers imple- 
mented a targeted intervention on 10% of eligible days, whereas others 
implemented the intervention on 100% of eligible days (Fabiano et al., 
2010; Owens et al., 2008). 

Researchers have yet to quantify the costs of these various implementation 
strategies, or their relative cost-effectiveness in producing desired outcomes. 
Given the documented variability across teachers using the same level of 
supports, researchers also need to consider variability in costs within pro- 
grams. Namely, variability in implementation is likely to result in inefficien- 
cies of dollars spent on the approach, as not all teachers adopt or benefit 
from each approach. This variability may also produce financial inefficiencies 
as it relates to student outcomes. That is, although costs of the intervention 
may be spread across several students, it may be that only a few students are 
experiencing beneficial effects. Thus, more information is needed regarding 
how variability in implementation impacts per student costs associated with 
the approach. 


Innovations in implementation strategies 


Given the documented variability in teachers’ implementation of classroom 
interventions (Fabiano et al., 2010; Owens et al., 2008) as well as the common 
barriers to implementation as reported by teachers (Collier-Meek, Sanetti, & 
Boyle, 2019; Long et al., 2016), researchers have recently been examining 
innovative adaptations to problem-solving consultation to enhance imple- 
mentation outcomes (i.e., adoption and implementation). Some adaptations 
have focused on enhancing individualization by including techniques asso- 
ciated with motivational interviewing and/or cognitive behavioral therapy 
(Collier-Meek, Sanetti, Levin, Kratochwill, & Boyle, 2019; Owens et al., 2017; 
Reinke et al., 2008) and by targeting mechanisms purported to affect imple- 
mentation outcomes (e.g., knowledge, motivation, self-efficacy, planning). 
Although it is possible that this individualization may prove to be more 
costly, the results from these trials suggest that these enhanced approaches 
are acceptable to teachers and may also be more effective for some teachers 
and their students than others. Other adaptations are leveraging interactive 
online technologies to target specific barriers to adoption and implementa- 
tion (e.g., time demands; intervention compatibility). Pilot studies of the use 
of interactive online technologies reveal that this strategy is acceptable to 
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a subset of teacher, as 30-50% can adopt and implement a classroom inter- 
vention with high quality with very minimal face-to-face supports, and can 
produce moderate to large change in student behaviors (Mixon et al., 2019; 
Owens et al., 2019). These results suggest that there may be some cost- 
efficiencies with the use of technology, as it may be more feasible to provide 
to a larger number of teachers. Although it is frequently assumed that 
leveraging technology is a cost-effective approach, a systematic review 
revealed mixed evidence for the cost-effectiveness of technology-based 
health-care programs (de la Torre-diez, Lépez-Coronado, Vaca, Aguado, & 
de Castro, 2015). This suggests that additional research is needed to inves- 
tigate the assumption that technology is cost-effective within the teacher 
consultation literature. 


The need for cost-analysis and cost-effectiveness studies 


The previously reviewed literature reveals the complex challenges that school 
administrators face in attempting to maximize the impact of their profes- 
sional development funds. One line of research that could facilitate data to 
effectively inform administrators’ decisions is that of economic evaluations. 
Cost-analysis research involves estimating the total costs of an intervention 
based on the total and component resources required to implement the 
intervention, or an ingredients-based approach (Belfield & Levin, 2013). 
We are unaware of any study that has described the relative costs of different 
implementation strategies. Thus, it may be difficult for administrators to 
know if approaches that have been categorized as best practices are a value 
within their professional development budget. When examining costs asso- 
ciated with different implementation strategies, it is important to account for 
variability in rates of implementation, as strategies that lead to minimal 
engagement may result in greater costs per student receiving the interven- 
tion. Thus, the total cost of the program should be considered both as costs 
distributed across the full sample (intent-to-treat approach) as well as dis- 
tributed across the participating subsample (completers approach). 
Cost-effectiveness analyses (CEA) compare the costs of two strategies to 
produce a single beneficial outcome (Crowley et al., 2018). Thus, CEA 
provide a ratio of the cost of a strategy per unit of effect (i.e., incremental 
cost-effectiveness ratio; ICER) relative to conditions as usual, and this 
approach can answer the question of which strategy requires more funds 
for the same effect. We are unaware of any studies that examine cost- 
effectiveness across different implementation strategies targeting teacher's 
implementation of classroom interventions. Claims are often made that 
technology can lead to efficiencies that are cost-effective; however, to our 
knowledge, this has not been tested in the context of consultation research. 
By understanding the relative costs and cost-effectiveness of different 
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implementation strategies, administrators may be able to maximize teacher 
and student outcomes in relation to dollars spent. 


Current study 


Using an ingredients-based approach (Belfield & Levin, 2013) within 
a societal perspective (i.e., net costs to society), we sought to conduct and 
economic evaluation of three implementation strategies that vary in focus 
and intensity (i.e., face-to-face standard consultation, face-to-face enhanced 
consultation, or interactive online supports), each of which were designed to 
promote teachers’ adoption and implementation of a daily report card (DRC) 
intervention. The standard condition represented best practice procedures in 
behavioral problem-solving consultation (Frank & Kratochwill, 2014); con- 
sultants refrained from using active ingredients of the enhanced condition. 
The enhanced condition adopted a behavioral problem-solving approach but 
included individualized techniques grounded in motivational interviewing 
and cognitive behavioral therapy to enhance teacher knowledge of behavioral 
principles and interventions; motivation, self-efficacy, and beliefs related to 
intervention adoption and implementation, and skills in classroom manage- 
ment. In the on-line condition, teachers were given a website that included 
brief video tutorials and an interactive “wizard” that was designed to mirror 
face-to-face consultation, guiding teachers in selecting target behaviors, 
tracking baseline data, setting goals, and modifying the goals to shape student 
behavior over time. See Table 1 for a visual display of the activities completed 
in each condition. 

We selected the DRC intervention because it (a) builds upon effective 
universal classroom management strategies, (b) is effective in changing 
a variety of behaviors (e.g., on-task, work productivity, following instruc- 
tions, remaining in seat/area, respect), including those associated with atten- 
tion-deficit hyperactivity disorder (ADHD) and oppositional and conduct 
problems (Pyle & Fabiano, 2017; Vannest, Davis, Davis, Mason, & Burke, 
2010), (c) is acceptable to teachers (Girio & Owens, 2009), and (d) can 
produce moderate to large effects within 2 months of implementation 
(Owens et al., 2012; Pyle & Fabiano, 2017). 

To achieve Aim 1 we conducted a cost-analysis using all consented 
teachers (intent-to-treat). To examine how variability in teacher implemen- 
tation affects intervention costs, we also estimated the costs of each approach 
using only the subset of teachers who actively engaged in the approach 
(completers; Aim 2). Lastly, we estimated the ICERs, or ratio of cost per 
effect, to examine the cost-effectiveness of each implementation strategy 
relative to the cost of typical practice (Aim 3). Sensitivity analyses were 
conducted to account for uncertainty present in cost estimates and enhance 
the robustness of our estimates. 
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Method 
Participants 


Data for the analyses were drawn from two studies (Owens et al., 2017, 2019). 
Participants in the first study were 58 teachers (each with one target student 
who received a DRC intervention) who were recruited in the fall of 2014 
from eight participating schools across two sites in the United States (28 
from Ohio and 30 from Florida) and randomly assigned to either the 
standard or enhanced consultation condition. Participants in the second 
study were 54 teachers (each with one target student who received a DRC 
intervention) who were recruited in the fall of 2017 from 10 elementary 
schools across two school districts in Alberta, Canada and participated in the 
on-line condition. 

Across both studies, participants reported an average of 13.2 years of 
teaching experience (SD = 8.1); there were no differences in teaching experi- 
ence reported by teachers in the two samples. In both studies, most teachers 
were women (Study 1: 93.1%; Study 2: 75.9%) who taught in general educa- 
tion classes (Study 1: 100%; Study 2: 85.2%). In Study 1, teachers were more 
likely to be teaching K-2 (Study 1: 51.8%; Study 2: 40.7%) and more likely to 
have reported having a Master’s degree (Study 1: 62.1%; Study 2: 18.5%). 

In Study 1, most target students were male (76%). Approximately half of 
the sample (54%) identified as Hispanic. Most students were from low to 
middle socioeconomic backgrounds (17.2% had a household income under 
15,000, USD 55.2% had an income between 15,000 and 49,999, USD 18.9% 
were above 50,000, USD 8.6% did not report income). Because the focus of 
Study 2, which examined the interactive online supports, was on teacher 
implementation behaviors, demographic information about target students 
was not obtained. 

With regard to school level data, in Study 1, the schools were diverse. The 
five Ohio schools had an average of 377 students per school, with 12%-29% 
of students receiving special education services and 35-75% receiving free or 
reduced lunch services. The three Florida schools had an average of 1,024 
students, with 4-11% receiving special education services and 76 - 95% 
receiving free or reduced lunch services. The average class size across sites 
ranged from 19 to 25 and the teacher was the sole educator in the room. In 
Study 2, the schools, the schools were also diverse, with an average of 473 
students per schools. Most schools (n = 6) served only elementary school 
students (K-6), but the others included middle school students. 

Consultants in Study 1 were post-doctoral fellows (n = 2), master’s level 
clinicians (n = 2) or graduate students in a master’s or doctoral program in 
psychology (n = 5). Six identified as Caucasian, one identified as African 
American, and two identified as Hispanic. To ensure integrity to each 
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consultation condition, facilitators attended a 3-day training and received 
weekly supervision throughout the trial. 


Recruitment 


All procedures were approved the research boards of all schools and uni- 
versities involved. Recruitment in both studies occurred by inviting all 
teachers in participating buildings to an initial 3-h workshop that included 
content on classroom management and DRC procedures (see details in 
Owens et al., 2017, 2019). Teachers consented to the project at this workshop. 
To participate, teachers were invited to identify a target student who would 
benefit from the DRC, and obtain parent consent for implementation of the 
intervention in the context of the research study. Student eligibility differed 
across the two studies. In Study 1, parents and students completed 
a comprehensive evaluation with the investigators to confirm presence or 
risk of ADHD (see details in Owens et al., 2017). In Study 2, students with 
elevated scores on the hyperactivity/inattention problems or conduct pro- 
blems subscales of the teacher-completed Strengths and _ Difficulties 
Questionnaire (SDQ; Goodman, 2001) were eligible (see Owens et al., 
2019). The priority in both studies was the evaluation of teacher implemen- 
tation behaviors. As such, the research team monitored some aspects of 
parent involvement but did not actively apply strategies to shape parent 
implementation behaviors. 


Implementation strategies 


Study 1 (standard and enhanced consultation conditions) 

Teachers in this study were randomized (see successful randomization pro- 
cedures in Owens et al., 2017) to either face-to-face standard consultation 
(n = 27) and face-to-face enhanced (n = 31); henceforth referred to as 
standard and enhanced. As can be seen from Table 1, teachers and consul- 
tants in both conditions participated in six events prior to launching the 
DRC: (1) the initial workshop, (2) a classroom management interview (3) 
a meeting to discuss and select target behaviors for the DRC (target behavior 
interview), (4) collection of baseline data, (5) a meeting to review baseline 
data and develop the DRC, and (6) a meeting with the child present to review 
and launch the DRC. Parents were invited to attend the launch meeting (48% 
attended in standard and 31% in enhanced). In both conditions, teachers 
were observed weekly, and met with the consultant on a biweekly basis (every 
other week) for up to eight consultation sessions. Details about the number, 
content, and duration of sessions were documented via consultation logs and 
consultation audio recordings. Teachers in each condition did not differ in 
the average number of sessions or average number of observations received 
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(see Owens et al., 2017). Between sessions, teachers tracked the daily fre- 
quency of student target behaviors as part of implementing the DRC. 
Biweekly consultation sessions involved review of the student’s progress on 
the DRC, review of data from the classroom observations, and feedback 
based on the observations. Teachers did not receive compensation for parti- 
cipating in consultation meetings. 

In the standard condition, the bi-weekly consultation sessions adopted 
a behavioral problem-solving approach with brief performance feedback. The 
guiding principles for this condition were that performance feedback should 
be limited to 5-10 min and unless the teacher initiated discussion of teacher 
values, beliefs or other content, the problem-solving remained child-focused. 
This condition was meant to mirror best practice procedures (Frank & 
Kratochwill, 2014) and refrain from using active ingredients of the enhanced 
condition. 

In the enhanced condition, motivational interviewing techniques were 
infused into each session (starting with a values interview during the class- 
room management interview). The bi-weekly consultation adopted 
a behavioral problem-solving approach but included techniques grounded 
in motivational interviewing and cognitive behavioral therapy to enhance 
teacher knowledge of behavioral principles and intervention (knowledge 
component), motivation, self-efficacy and beliefs related to intervention 
adoption and implementation (beliefs component), and skills in classroom 
management (skills component). Performance feedback was enhanced by 
focusing on the connection between teacher implementation behaviors and 
student outcomes and on discrepancies between teacher beliefs and behaviors 
(see Owens et al., 2017 for details). 

Parents were invited to attend the DRC launch meeting. Some parents 
requested additional meetings with the consultant to discuss parenting 
related to the DRC. The teacher was not present at these meetings. 


Study 2 (on-line condition) 

This study included one condition, the interactive online supports condition. 
During the initial teacher workshop, the DRC was introduced and teachers 
were shown how to use the interactive website, referred to as the Daily 
Report Card.Online (DRC.O). The DRC.O website was developed to provide 
educators with an accessible, interactive, web-based platform to aid in the 
development and implementation of a DRC intervention (Mixon et al., 2019; 
Owens et al., 2019). The interactive DRC Wizard was designed to mirror 
face-to-face consultation, guiding teachers in selecting target behaviors, 
tracking baseline data, setting goals, and modifying the goals to shape student 
behavior over time. The website included video models, a graphing resource, 
downloadable resources, and a “Contact us” portal. If teachers had questions, 
a consultant would reply in 24 to 48 hours. There was no face-to-face 
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consultation provided by the investigative team. Once teachers began a DRC, 
they were encouraged to implement the DRC and enter daily data for 2 
months. To facilitate this, the research team sent e-mail prompts at specific 
points of implementation (e.g., after completing a pre-intervention SDQ, at 
DRC launch) and/or after a lapse in data entry. Details about the number and 
duration of website visits by teachers were documented via website analytics. 
Details about e-mail communication with teachers were documented via 
consultation logs. 


Costs 


Consistent with recommendations for conducting economic evaluations of 
intervention programs (Crowley et al., 2018), a societal perspective was taken 
for calculation of costs, with direct costs to the schools disaggregated. All 
direct costs were calculated for the time period including the initial workshop 
and the first 2 months of intervention delivery. In order to estimate costs 
associated with the implementation strategies, costs associated with the 
research components of the studies, such as research staff time and costs 
associated with administering the study instruments, were excluded. A two- 
month timeframe was selected given the evidence that the largest change in 
target behaviors occurs within the first 2 months of intervention (Holdaway 
et al., 2018; Owens et al., 2012). 

Using workshop attendance and consultation logs from each study and 
website analytics from the second study, we calculated costs associated with 
(a) materials (used in the workshop and throughout implementation), (b) 
teachers’ time (in the workshop, in consultation, on-line, and while data 
tracking), (c) consultants’ time (in the 3-day training, in consultation, in 
observation, and meeting with families), and (d) parents’ time in meetings. 
With regard to materials, the CostOut tool available through Columbia 
University’s Center for Benefit-Cost Studies of Education was used to esti- 
mate costs of materials used in the workshop, consultation, and intervention 
delivery, including workshop materials and paper for bi-weekly newsletters 
distributed in the enhanced consultation. The cost of access to DRC.O was 
estimated to be 120 USD per student, consistent with the current payment 
package for this service. This is a conservative estimate, as this price point is 
the highest possible cost per student, given the flexible payment packages 
forthcoming for the DRC.O. 

With regard to costs associated with teachers’ and consultants’ time, costs 
were calculated using the U.S. Bureau of Labor Statistics mean annual wages 
for these respective positions. The wage for teachers was estimated from the 
mean salary for teachers in 2018 ($62,200; 2019a), and this value was divided 
by 1,600, which assumes teachers receive a nine-month salary for 40-h weeks, 
resulting in an hourly rate of 38.88. USD The mean hourly wage of a school 
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psychologist ($37.97; 2019a) was used to estimate costs attributed to the 
consultant. Opportunity costs related to parent time in meetings were calcu- 
lated using the mean hourly wage across all industries ($27.23; 2019b). 
Wages were assumed to be equivalent across sites and studies in order for 
comparison, despite the regional differences in wages and costs of living. 
Although our sample includes international locations, all costs were esti- 
mated from publicly available data sources and were adjusted to 
2018 USS. dollars. 

For costs related to teacher time using DRC.O implementation strategy, 
the website was programmed to collect data on teachers’ time spent across 
various features. This feature allowed teachers to leave the website open while 
attending to other tasks, resulting in multiple outliers in time spent on the 
webpages. Outliers were identified using the Median Absolute Deviation 
approach (Leys, Ley, Klein, Bernard, & Licata, 2013) and identified outliers 
were Winsorized, or replaced by a value at the edge of the distribution. In the 
online condition, consultation was delivered via e-mail rather than in-person 
meetings. A log of all e-mail exchanges was maintained and, given evidence 
of the brevity of e-mail exchanges (Litmus, 2019; Ye, Rust, Fry-Johnson, & 
Strothers, 2010), it was estimated that time spent reading e-mails was less 
than 1 minute, and time spent typing an e-mail was approximately 
3.5 minutes. 

These above-described costs were calculated and summed for each child. 
For each implementation strategy, costs were estimated as total costs from 
a societal perspective, as well as direct costs to the school. Direct costs to the 
school were estimated as the sum of in-person training and material costs, 
assuming that other costs of consultation and implementation attributed to 
teacher or consultant time would be within the context of their school day 
duties as a teacher and school psychologist. Thus, their time during these 
tasks would not be monetized beyond what a school is already paying for 
typical practice. 


Student outcomes 


Student outcomes were defined as the change in student target behaviors. 
With consultation, teachers selected target behaviors (e.g., interruptions, out 
of seat) and were instructed to record the frequencies of the target behavior 
for at least three school days to establish a baseline. Teachers were then 
instructed to document the occurrence of each child’s target behaviors daily. 
The magnitude of students’ behavioral improvements were measured by 
single-case design effect sizes. For each DRC target behavior, the number 
of data points within each of the 2 months of intervention was calculated. If 
there were at least 3 days of baseline data and 10 days of data in the month, 
the change during that month of the intervention was quantified by 
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calculating Tau-U and Taunonoverlap (Parker, Vannest, Davis, & Sauber, 2011), 
a single-case effect size that accounts for possible trends during baseline. The 
TaUnonoverlap Was used to calculate the effect unless the baseline tau value was 
>10, in which case Tau-U was used to correct for the baseline trend. 


Analytic plan 


Cost estimates 

Costs were estimated as a sum of each implementation strategy, the average 
cost per participant, and the marginal cost. Marginal costs are an estimate of 
the cost of providing the intervention to one additional student in the class- 
room. Marginal costs reflect the variable costs of the program or costs 
attributable to each participant (Crowley et al., 2018). Thus, given that the 
cost to train a teacher and provide the teacher with materials for the inter- 
vention would not vary based on the number of students served, marginal 
costs were calculated as the average cost per student, minus training and 
material costs. For Aim 1, costs per student were calculated as the total cost 
of the condition divided by the total sample, consistent with the intent-to- 
treat approach. For Aim 2, the total cost of the implementation strategy was 
divided by the number of completers, defined as the number of students who 
had data entered for at least 10 days in either the first or second month of the 
intervention. Thus, 13 students were considered completers for standard 
consultation (48% of sample), 21 students were completers for enhanced 
consultation (68% of sample), and 25 students were completers for online 
condition (46% of sample). This threshold (10 of 20 school days in a month; 
or 50% compliance) was selected because it is consistent with methods used 
in previous studies (Owens et al., 2012) and because there is some evidence 
that a 51% compliance threshold represents a minimum threshold needed to 
change student classroom behavior (Owens et al., 2020). 


Cost-effectiveness estimates 

To calculate the cost-effectiveness of each implementation strategy relative to 
typical practice conditions (Aim 3), the estimated average cost per student 
was divided by the average effect size per student. A small portion of students 
(n = 16) had inadequate Month 2 data to calculate the Tau effect size, and for 
these students, Month 1 Tau effect sizes were imputed to Month 2 effects 
given evidence that Month 1 effects tend to remain stable or even increase 
into Month 2 most students (Owens et al., 2012). Given that Tau values 
between.20 and .60 represent a moderate effect and available DRC bench- 
marks for response to treatment at 2 months fall within this range (e.g., 
Holdaway et al., 2018), incremental cost-effectiveness ratios (ICER) are pre- 
sented as ratios of the cost per .40 Tau effect size. 
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An ICER represents the average cost of the implementation strategy 
beyond that of typical practice, divided by the average effectiveness. The 
following formula was used to calculate the ICERs presented in this study: 
\openup $${\rm{ICER = }}\left( {{{\rm{C}}_{\rm{A}}}{\rm{- }}{{\rm{C}}_ 
{\rm{B}}}} \right){\rm{/}}\left( {\left( {{\rm{Tau effect size}}} \right){\rm{*}} 
{\rm{.40}}} \right)$$ 

where Ca is the average cost of the consultation package in question per 
student, Cx is the cost of treatment as usual, the Tau effect size is the average 
improvement in target behaviors relative to baseline, and the effect size is 
multiplied by .40 to allow the ICER to be interpreted as a moderate improve- 
ment per student. Thus, the ICER can be interpreted as the per-student cost 
of the implementation strategy required in order to achieve a moderate 
improvement in the DRC target behaviors. Thus, lower ICERs indicate that 
the approach costs less for the desired effect. Of note, the ICERs are equiva- 
lent between the overall sample and completers, given the 1:1 ratio by which 
non-completers decrease the per student cost to the degree by which they 
decrease the per student effect sizes. ICERs were not calculated between 
implementation strategies given that randomization to condition only 
occurred for two of the three conditions (i.e., standard and enhanced con- 
sultation in Study 1). Therefore, cost-effectiveness of each condition was 
estimated as relative costs and effects compared to typical practice, rather 
than relative to each other. 


Sensitivity analyses 

Because cost estimates are based on a variety of assumptions, current best 
practice standards for cost-effectiveness analyses include estimation of varia- 
bility in the point estimates based on these assumptions (Crowley et al., 
2018). Therefore, a series of sensitivity analyses were conducted in order to 
estimate the extent to which estimated ICERs were sensitive to variability in 
the model parameters, assumptions, or outliers. In the first sensitivity analy- 
sis, we examined the extent to which estimated costs and effects were driven 
by a small portion of extreme-cost cases. Thus, in subsequent analyses, the 
top 10%, then bottom 10% of students in terms of total cost were dropped 
from each condition. In the second sensitivity analysis, we examined the 
impact of wage variations on cost estimates by substituting the wage esti- 
mates with the 10th and 90th percentile wages for teachers and school 
psychologists as reported by the Bureau of Labor Statistics. Finally, 
a sensitivity analysis was conducted to assess for variations in the level of 
parent involvement. Thus, we calculated an estimate of per student costs if 
parents were not involved in the intervention and if parents were involved in 
each potential component (i.e., launch meeting, and three 1-h consultation 
meetings including both parent and consultant costs). For all comparisons 
described, independent samples t-tests were calculated to determine the 
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statistical significance of observed differences in costs of the implementation 
strategies. The Bonferroni correction was applied to account for familywise 
error, resulting in an alpha of .004. 


Results 
Aim 1: costs of implementation strategies 


Table 2 presents the cost per student for each component of the implementa- 
tion strategy, the total cost per student, and the direct costs to the school per 
student (i.e., costs of in-person training and materials). In the full sample, the 
online system was less costly than either in-person approach. In terms of 
component costs, the online system had a unique advantage of requiring 
minimal consultant costs relative to the other two strategies. The most 
expensive component of the online system was the materials cost of using 
the DRC.O (i.e., 120 USD). When examining the direct cost to the school, the 
online system remained the least costly implementation strategy, followed by 
standard consultation and enhanced consultation which remained the most 
costly. 


Aim 2: costs of implementation strategies for completers 


To achieve Aim 2 we examined the costs of the intervention per completer, 
defined as a student who had at least 10 days of data tracked in their DRC 
during either of the first 2 months. Thus, we re-distributed the total costs of 
each implementation strategy across only those students who received at least 
a minimal amount of the intervention. Table 2 displays the costs of each 
implementation strategy by component per student who received the service. 
Unlike in Aim 1 analyses, standard consultation was the most expensive per 
completer, both overall and when examining direct costs to schools. The 
online system remained the least expensive per student. 


Aim 3: cost-effectiveness of implementation strategies 


For Aim 3, we calculated the ICERs, or cost per effect, of each implementa- 
tion strategy relative to typical practice. Table 3 displays the ICERs per 
implementation strategy, presented as the cost per student in order to achieve 
a moderate response to intervention (Tau = .40). From the societal perspec- 
tive, the standard condition costs 2,042.10 USD per student beyond typical 
practice per student to achieve moderate improvement in student target 
behaviors. Alternatively, the cost of the enhanced approach was 1,451.70 
USD per student to achieve moderate improvement in each student’s target 
behaviors. When considering the costs and effects relative to typical practice, 
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the online approach was the most cost-effective among the three strategies, 
with a societal cost of 719.33 USD per student to achieve a moderate 
improvement in target behaviors. 

The ICERs presented in Table 3 are adjusted to be interpreted as the cost 
to achieve a moderate improvement per student in the respective condition. 
The portion of students achieving this moderate response to intervention by 
the second month was similar across groups, with 7 (26%) students in the 
enhanced condition, 9 (29%) students in the enhanced condition, and 12 
(22%) students in the online condition achieving a moderate improvement 
(Tau = .40). 


Sensitivity analyses 

Table 4 displays the results of the sensitivity analyses conducted to account for 
variability in cost estimates related to the assumptions made in the cost 
analyses. In order to examine whether group differences were driven by 
a small number of extreme-cost cases, the top 10% in terms of costs were 
dropped from each implementation strategy and the ICERs were calculated 
again. This procedure was repeated for the bottom 10% of cases. Similar to the 
main analyses, removal of the most extreme cases demonstrated that online 
consultation was the most cost-effective option, followed by enhanced consul- 
tation, followed by standard consultation. Removing the most costly cases did 
not result in significantly different cost or effect estimates across implementa- 
tion strategies. 

As can be seen in Table 4, if teacher and consultant wages are estimated at 
the 10th percentile, each implementation strategy would be significantly less 
costly than if teacher and consultant wages are at the mean. Similarly, if 
teacher and consultant wages are at the 90th percentile, each strategy would 
be significantly more costly than if teacher and consultant wages are at the 
mean. Regardless of the wage estimate used, conclusions regarding the cost- 
effectiveness across the three implementation strategies are consistent, such 
that the provision of the online consultation via DRC.O is the most cost- 
effective, followed by enhanced consultation, with standard consultation 
being the least cost-effective option. 

Table 4 displays results of the sensitivity analysis examining the impact of 
parent involvement on cost estimates. Cost estimates are not significantly 
different from our estimates if parent involvement is assumed to be minimal. 
Each consultation package would be significantly more costly if parents were 
involved in the launch meeting and attended three 1-h consultation sessions. 
The on-line condition remained the least costly option if parents were 
assumed to be maximally involved. 
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Discussion 


Using an ingredients-based approach and a societal perspective, we estimated 
the costs of three implementation strategies (for the initial workshop and 
across 2 months of implementation) and their cost-effectiveness relative to 
typical practice. In terms of costs beyond those of typical practice, the results 
reveal that the interactive online implementation strategy was the least 
expensive overall ($306.89 per student), followed by standard consultation 
($634.00 per student), with enhanced consultation costing the most per 
student ($863.78 per student). When considering these costs in relation to 
the students’ improvements in behavior, the costs for the desired effects were 
lowest for online consultation, followed by enhanced consultation, and, 
further followed by standard consultation. The ICERs reveal that (a) the 
online approach costs the least for the desired effect relative to the face-to- 
face conditions, and (b) although more costly, enhanced individualized 
strategies may produce a greater effect than standard consultation character- 
ized by a one-size-fits-all approach. These findings advance our knowledge 
about the costs of implementation strategies and offer support for some 
assumptions about technology-assisted supports and enhanced individualized 
consultation. 

First, to our knowledge this is the first study to put a price on various 
implementation strategies. Prior studies have revealed the costs of interven- 
tion packages per youth at-risk of ADHD, including a 2-month trial of high 
quality medication management ($2,022 or 2,230 USD in 2019 dollars), or 
a 2-month trial of school-based behavior therapy ($1,706 or 1,881 USD in 
2019 dollars; Page et al., 2016), yet this is the first study to demonstrate the 
potential cost-savings utility of consultation for school-based interventions. 
Given the limited use of targeted classroom interventions by elementary 
school teachers (e.g., Martinussin et al., 2011) and the high variability in 
implementation when used (e.g., Fabiano et al., 2010; Owens et al., 2008), 
finding effective, feasible, and affordable implementation strategies is critical 
to ensuring that students receive needed intervention. Our findings allow 
school administrators to consider the extent to which the three approaches 
used in these studies fit within their school budgets. Considering the high 
annual societal costs associated with special education services per student 
with ADHD ($5,007, or 5,842 USD in 2019 dollars; Robb et al., 2011), future 
research should examine the extent to which the costs of these implementa- 
tion strategies may reduce the need for, and thus costs of, special education 
for some students. 

Second, we found support for two important hypotheses. Namely, the indi- 
vidualized enhanced strategy was more costly, yet students whose teachers used 
this strategy experienced the greatest effects. Thus, in the enhanced consultation 
strategy, increased costs were associated with increased positive effects (i.e., 
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schools may get their money’s worth). This is an important finding given the 
number of enhanced approaches that have been evaluated in recent years (e.g., 
Bradshaw et al., 2018; Collier-Meek et al., 2019; Conroy et al., 2015). With 
further study of these approaches, we may be able to identify the mechanisms 
through which each implementation strategy operates to produce the desired 
implementation outcome (Lewis et al., 2018) and further streamline how we 
train consultants in the approach. Similarly, we found support for the claim that 
technology can lead to efficiencies that are cost-effective, as this produced the 
smallest ICER, or the per-student cost required to achieve a moderate improve- 
ment in the DRC target behaviors. These conclusions regarding cost- 
effectiveness did not change with the inclusion of the sensitivity analysis, 
suggesting that these findings are robust against variations in costs attributable 
to regional or staffing differences. Given this finding, cost-effectiveness analyses 
of the use of other technologies in consultation (e.g., bug-in-the-ear and Swivel 
cameras; Grygas Coogle, Ottley, Rahn, & Storie, 2018) and video-conference 
technologies (e.g., Schultz et al., 2018) are warranted to determine which 
technologies provide cost-effective implementation strategies. 

These findings, coupled with the emerging evidence that individually tailored 
approaches may produce benefits for some teachers (i.e., those with barriers to 
integrity) but may be unnecessary for other teachers (e.g., teachers who can 
leverage technology; those without barriers to integrity), suggests that additional 
research on individualized approaches and on the use of technology are fruitful 
pursuits. Indeed, school administrators may find cost efficiencies in matching 
implementation strategies to teacher needs, as being able to offer technology to 
those who can achieve effects without expensive face-to-face consultation and 
reserving the more costly enhanced consultation to those who need such support 
can maximize outcomes for dollars spent. Thus, it may be profitable for research- 
ers to examine possible moderators of cost-effectiveness (e.g., teacher implemen- 
tation integrity) and devote efforts toward the development of reliable and valid 
tools for measuring teacher characteristics that may predict implementation 
integrity and/or specific professional development needs (Owens, Allan, Hustus, 
& Erchul, 2018; Owens et al., 2017). Such research could facilitate the identifica- 
tion of teacher characteristics that may moderate response to consultation and/or 
prioritize groups of teachers for specific types of support. Such specificity could 
help school personnel expend consultative resources efficiently. 

It is important to note that we calculated ICERs to represent a ratio of the 
cost per .40 Tau effect size, a moderate effect size. Finding a meaningful 
outcome in cost-effectiveness studies can be a challenge. We selected this 
outcome, given the evidence that this benchmark can be expected after 1 and 
2 months of intervention (Holdaway et al., 2018; Owens et al., 2012). Further, 
using information from other studies (Mixon et al., 2019; Owens et al., 2012, 
2019), these effect sizes represent (a) change in student behaviors that 
teachers view as a top priority, (b) changes that move students from the 
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clinically significant range to near normative range on teacher rating scales, 
and (c) 40% to 60% reductions in negative behaviors relative to baseline 
levels. Although effect sizes should be interpreted in the context of student 
needs, goals, history, and context (Vannest & Ninci, 2015), we believe that 
this represents a meaningful and interpretable outcome. Nonetheless, addi- 
tional research is needed to identify outcomes that can be used across studies 
to enhance the interpretability of ICERS. 

Lastly, parent involvement in the current implementation strategies was 
relatively limited, with few parents engaging in offered meetings. Given the 
priorities of the studies from which our data were drawn, data on parent 
involvement were limited in the on-line condition. Our sensitivity analysis 
demonstrated that costs of each strategy would be significantly greater if parents 
were more involved in implementation. Given that greater parent involvement in 
school-based interventions has been associated with improved effects (Vannest 
et al., 2010), research is needed to examine whether these increased costs impact 
the cost-effectiveness of these interventions. Further, cost-effectiveness estimates 
may be even more favorable under different circumstances and/or with techno- 
logical advances that facilitate more parent involvement. 


Limitations 


These findings should be considered within the context of the study’s limita- 
tions. First, the current study used two separate samples to estimate costs and 
effects of the three implementation strategies. This lack of randomization 
precluded our ability to make direct comparisons between each condition, 
and therefore all cost-effectiveness indices represent the condition’s comparison 
to typical practice. Future studies are warranted to utilize randomization 
processes in order to directly compare across implementation strategies and 
eliminate concerns related to potential inherent group differences. Second, all 
costs related to wage are estimates based on U.S. estimates from publicly 
available resources. These estimates may not be representative of different 
staffing decisions (e.g., having a school counselor rather than a school psychol- 
ogist) or regional variations in wages, and therefore calculated costs should be 
interpreted as estimates that are likely to vary in real-world settings. 
Additionally, these estimates were calculated using U.S. wage estimates, 
although a portion of the sample was from Canada. The multi-national nature 
of our sample serves as a strength, yet the current cost estimates may not 
generalize across countries. Although detailed time estimates from timesheets 
were used to estimate wages, e-mail logs did not include estimates for time 
spent on e-mails. Estimates for time in consultation e-mails were best-estimates 
based on currently available data regarding time spent on e-mails and may not 
accurately represent the actual time spent in e-mails. However, each consultant 
and teacher would have needed to spend an additional 3.5 hours on e-mails per 
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student in the online condition in order to make it more costly than other 
conditions. Given this high estimate, it is unlikely that potential variations in 
these estimates would impact our conclusions. As with many cost-estimate 
studies, some costs related to the consultation may not have been captured, 
such as wage estimates related to systems-level administrative support, infor- 
mational technology support staff time, overhead costs, or travel time if teachers 
or consultants need to travel to the trainings or consultant meetings. These 
costs may be important considerations for the overall societal costs, yet they are 
unlikely to change the direct costs to schools. Lastly, best practices for single 
subject research (Kratochwill et al., 2010), recommend having 5 days of baseline 
data. Given the challenges to meeting this standard in practice, we relaxed this 
criterion to maximize our sample size. 


Conclusions 


Our findings support the assumptions related to the cost-effectiveness advan- 
tages of technology-based interventions. The findings also suggest that there 
may be substantial cost-effectiveness advantages to prescribing specific stra- 
tegies to teachers based on their likely responsiveness. Enhanced face-to-face 
consultation appears to be an effective and expensive approach to helping 
teachers implement a DRC. Thus, identifying teacher characteristics that 
moderate response to type of consultation will be an important next step 
in this line of research. 
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